Metadata-Version: 2.1
Name: yival
Version: 0.1.0a8
Summary: YiVal is an open-source project designed to revolutionize the way developers and researchers evaluate and refine AI models.
License: Apachev2
Author: Tao Feng
Author-email: taofeng.nju@gmail.com
Requires-Python: >=3.10,<3.13
Classifier: License :: Other/Proprietary License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Provides-Extra: trainers
Requires-Dist: aiohttp-socks (>=0.8.0,<0.9.0)
Requires-Dist: alpaca-eval (>=0.3.1,<0.4.0)
Requires-Dist: bert-score (>=0.3.13,<0.4.0)
Requires-Dist: bitsandbytes (>=0.41.1,<0.42.0) ; extra == "trainers"
Requires-Dist: dash (>=2.11.1,<3.0.0)
Requires-Dist: dash-bootstrap-components (>=1.4.2,<2.0.0)
Requires-Dist: dash-dangerously-set-inner-html (>=0.0.2,<0.0.3)
Requires-Dist: faiss-cpu (>=1.7.4,<2.0.0)
Requires-Dist: fuzzywuzzy (>=0.18.0,<0.19.0)
Requires-Dist: hydra-core (>=1.3.2,<2.0.0)
Requires-Dist: langchain (>=0.0.275,<0.0.276)
Requires-Dist: litellm (==0.1.607)
Requires-Dist: numpy (>=1.25.2,<2.0.0)
Requires-Dist: omegaconf (>=2.3.0,<3.0.0)
Requires-Dist: openai (==0.27.10)
Requires-Dist: pandas (>=2.0.3,<3.0.0)
Requires-Dist: pandas-stubs (>=2.1.1.230928,<3.0.0.0)
Requires-Dist: peft (>=0.5.0,<0.6.0) ; extra == "trainers"
Requires-Dist: pillow (==9.4.0)
Requires-Dist: psutil (>=5.9.5,<6.0.0)
Requires-Dist: pydantic (>=2.1.1,<3.0.0)
Requires-Dist: pydoc-markdown (>=4.8.2,<5.0.0)
Requires-Dist: pyngrok (>=6.0.0,<7.0.0)
Requires-Dist: python-levenshtein (>=0.21.1,<0.22.0)
Requires-Dist: replicate (>=0.12.0,<0.13.0)
Requires-Dist: rich (>=13.4.2,<14.0.0)
Requires-Dist: rouge (>=1.0.1,<2.0.0)
Requires-Dist: scipy (>=1.11.3,<2.0.0) ; extra == "trainers"
Requires-Dist: streamlit (>=1.27.2,<2.0.0)
Requires-Dist: tiktoken (>=0.4.0,<0.5.0)
Requires-Dist: torch (>=2.0.0,!=2.0.1,!=2.1.0)
Requires-Dist: tqdm (>=4.65.0,<5.0.0)
Requires-Dist: transformers (>=4.33.0,<5.0.0)
Requires-Dist: trl (>=0.7.1,<0.8.0) ; extra == "trainers"
Requires-Dist: types-pillow (>=10.0.0.2,<11.0.0.0)
Requires-Dist: types-pyyaml (>=6.0.12.11,<7.0.0.0)
Requires-Dist: types-requests (>=2.31.0.2,<3.0.0.0)
Requires-Dist: types-tqdm (>=4.65.0.2,<5.0.0.0)
Description-Content-Type: text/markdown

<!-- markdownlint-disable MD033 -->
<!-- markdownlint-disable MD041 -->

<p align="center">
    <h1 align="center">
        <img src="https://github.com/YiVal/YiVal/assets/1544154/b0c681e7-7474-4b87-9c69-fde6e0e47401"
         alt="YiVal Logo" width="100"
        height="100" style="vertical-align: middle;">
        YiVal
    </h1>
     <p align="center">⚡ Build any Generative AI application with evaluation
    and enhancement ⚡</p>
</p>

<!-- markdownlint-disable-next-line MD013 -->
👉 Follow us: [![Twitter](https://img.shields.io/twitter/url/https/twitter.com/YiValai.svg?style=social&label=Follow%20%40YiVal)](https://twitter.com/yivalloveaigc) |
[![Discord](https://dcbadge.vercel.app/api/server/HnUWVW4kth?compact=true&style=flat)](https://discord.gg/HnUWVW4kth)

[![Downloads](https://static.pepy.tech/badge/YiVal/month)](https://pepy.tech/project/YiVal)
[![License: MIT](https://img.shields.io/badge/License-Apache2.0-yellow.svg)](https://opensource.org/licenses/MIT)
[![GitHub star chart](https://img.shields.io/github/stars/YiVal/YiVal?style=social)](https://star-history.com/#YiVal/YiVal)
[![Open Issues](https://img.shields.io/github/issues-raw/YiVal/YiVal)](https://github.com/YiVal/YiVal/issues)

## What is YiVal?

**YiVal is a versatile platform and framework that streamlines the evaluation
and enhancement of your Generative AI applications.**  
It empowers you to
generate better results, reduce latency, and decrease inference cost easily.
Depending on your knowledge and comfort level, YiVal will help you
simultaneously optimize prompts, model metadata, model parameters, and
retrieval configurations. You can easily customize your test data, evaluation
methods, and enhancement strategies, all in one place. Enhance and evaluate
everything with ease!

[Check out our quickstart guide!][1]

[1]: https://github.com/YiVal/YiVal/blob/master/demo/tutorial_notebook/tutorial.md

<img src="https://github.com/YiVal/YiVal/assets/1544154/dba5acd9-995c-45fd-9d08-c7cf198a77ad">

## Installation

### Prerequisites

- **Python Version**: Ensure you have `Python 3.10` or later installed.
- **OpenAI API Key**: Obtain an API key from OpenAI. Once you have the key, set
it as an environment variable named `OPENAI_API_KEY`.

### Installation Methods

#### Using pip (Recommended for Users)

Install the `yival` package directly using pip:

```bash
pip install yival
```

#### Development Setup Using Poetry

If you're looking to contribute or set up a development environment:

1. **Install Poetry**: If you haven't already, [install Poetry](https://python-poetry.org/docs/#installation).

2. **Clone the Repository**:

    ```bash
    git clone https://github.com/YiVal/YiVal.git
    cd YiVal
    ```

3. **Setup with Poetry**: Initialize the Python virtual environment and install
  dependencies using Poetry. Make sure to run the below cmd in `/YiVal` directory:

    ```bash
    poetry install --sync
    ```

## Trying Out YiVal

After setting up, you can quickly get started with YiVal by generating datasets
of random tech startup business names.

### Steps to Run Your First YiVal Program

1. **Navigate to the yival Directory**:

    ```bash
    cd /YiVal/src/yival
    ```

2. **Set OpenAI API Key**: Replace `$YOUR_OPENAI_API_KEY` with your
  actual OpenAI API key.

    ```bash
    export OPENAI_API_KEY=$YOUR_OPENAI_API_KEY
    ```

3. **Define YiVal Configuration**:
   Create a configuration file named `config_data_generation.yml` for automated
   test dataset generation with the following content:

    ```yaml
    description: Generate test data
    dataset:
      data_generators:
        openai_prompt_data_generator:
          chunk_size: 100000
          diversify: true
          model_name: gpt-4
          input_function:
            description: # Description of the function
              Given a tech startup business, generate a corresponding landing
              page headline
            name: headline_generation_for_business
            parameters:
              tech_startup_business: str # Parameter name and type
          number_of_examples: 3
          output_csv_path: generated_examples.csv
      source_type: machine_generated
    ```

4. **Execute YiVal**:
   Run the following command from within the `/YiVal/src/yival` directory:

    ```bash
    yival run config_data_generation.yml
    ```

5. **Check the Generated Dataset**:
   The generated test dataset will be stored in `generated_examples.csv`.

## Demo

[Demo Video](https://github.com/YiVal/YiVal/assets/1544154/83efc730-9a89-4da6-81be-15b0d30325a2)

| Use Case Demo | Supported Features                                                                                                                                                                                                                                                                                                                                                                                                                                                             | Colab Link |
|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------|
| 🐯 Craft your AI story with ChatGPT and MidJourney | **Multi-modal** support: Design an AI-powered narrative using YiVal's multi-modal support of simultaneous text and images. It supports native and seamless [Reinforcement Learning from Human Feedback](https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback)(RLHF) and [Reinforcement Learning from AI Feedback](https://arxiv.org/abs/2309.00267)(RLAIF). Please watch the video above for this use case.                                                |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1DgtDZghleiLEaaNF7f4vSGJ4ChDVls2X?usp=sharing) |
| 🌟 Evaluate performance of multiple LLMs with your own Q&A test dataset | Conveniently **evaluate and compare** performance of your model of choice against 100+ models, thanks to [LiteLLM](https://github.com/BerriAI/litellm). Analyze model performance benchmarks tailored to your **customized test data** or use case.                                                                                                                                                                                                                            |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1cSjsEKNwDq8W4O2oeRI93vH-BhYH5JXj?usp=sharing) |
| 🔥 Startup Company Headline Generation Bot | Streamline generation of headlines for your startup with automated test data **creation**, prompt **crafting**, results **evaluation**, and performance **enhancement** via GPT-4.                                                                                                                                                                                                                                                                                             |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1EiWUL8rE_kfNLXVPowCWCh6hwHFagvs_?usp=sharing) |
| 🧳 Build a Customized Travel Guide Bot | Leverage **automated prompts** inspired by the travel **community's** most popular suggestions, such as those from [awesome-chatgpt-prompts](https://github.com/f/awesome-chatgpt-prompts).                                                                                                                                                                                                                                                                                    |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1L7miRwTQSZfm5xOKBakWOG5bOumMynpv?usp=sharing) |
| 📖 Build a Cheaper Translator: Use GPT-4 to teach Llama2 to create a translator with lower inference cost | Using [Replicate](https://replicate.com/docs/guides/fine-tune-a-language-model) and GPT-4's test data, you can **fine-tune** Llama2's translation bot. Benefit from 18x savings while experiencing only a 6% performance decrease.                                                                                                                                                                                                                                           |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](<https://colab.research.google.com/drive/1HtQOadV0o3vrXjVI9Nf0Xv4rLarqu-fv?usp=sharing>) |
| 🤖️ Chat with Your Favorite Characters - 澹台烬(Dantan Ji) from《长月烬明》(Till the End of the Moon) | Bring your favorite characters to life through automated prompt creation and **character script retrieval**.                                                                                                                                                                                                                                                                                                                                                                   |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/12XS1fhgxRAHsRZPXtJ8c5exslgj7EDHE#scrollTo=31Wq1Oeb-bSY) |
| 🔍Evaluate guardrails's performance in generating Python(.py) outputs | [Guardrails](https://github.com/ShreyaR/guardrails): where are my guardrails? 😭 <br>Yival: I am here. ⭐️<br><br>The **integrated evaluation** [experiment](https://github.com/ShreyaR/guardrails/issues/345) is carried out with 80 LeetCode problems in csv, using guardrail and using only GPT-4. The accuracy drops from 0.625 to 0.55 with guardrail, latency increases by 44%, and cost increases by 140%. Guardrail still has a long way to go from demo to production. | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](<https://colab.research.google.com/drive/1QgRQmFmC_L07Ler4vbq_vcCNm_OHmJL_#scrollTo=KnmaSTEc13Rg>) |

## Contribution Guidelines

If you want to contribute to YiVal, be sure to review the [contribution guidelines](https://yival.github.io/YiVal/contributing/).
We use [GitHub issues](https://github.com/YiVal/YiVal/issues) for tracking
requests and bugs.
Please join [YiVal's discord channel](https://discord.gg/HnUWVW4kth) for
general questions and discussion.
Join our collaborative community where your unique expertise as researchers and
software engineers is highly valued! Contribute to our project and be a part of
an innovative space where every line of code and research insight actively
fuels advancements in technology, fostering a future that is intelligently
connected and universally accessible.

## Contributors

<a href="https://github.com/YiVal/YiVal/graphs/contributors">
  <img src="https://contrib.rocks/image?repo=YiVal/YiVal" />
</a>  
<p align="center">
<br>
🌟 YiVal welcomes your contributions! 🌟<p align="center">
🥳 Thanks so much to all of our amazing contributors 🥳</p>

## Paper / Algorithm Implementation

| **Paper** | **Author** | **Topics** | **YiVal Contributor** | **Data Generator** | **Variation Generator** | **Evaluator** | **Selector** | **Evolver** | **Config** |
|---|---|---|---|---|---|---|---|---|---|
| [Large Language Models Are Human-Level Prompt Engineers](https://arxiv.org/abs/2211.01910) | [Yongchao Zhou](https://arxiv.org/search/cs?searchtype=author&query=Zhou,+Y), [Andrei Ioan Muresanu](https://arxiv.org/search/cs?searchtype=author&query=Muresanu,+A+I), [Ziwen Han](https://arxiv.org/search/cs?searchtype=author&query=Han,+Z) | YiVal Evolver, Auto-Prompting | [@Tao Feng](https://github.com/oliverfeng) | [OpenAIPromptDataGenerator](https://github.com/YiVal/YiVal/blob/master/src/yival/data_generators/openai_prompt_data_generator.py) | [OpenAIPromptVariationGenerator](https://github.com/YiVal/YiVal/blob/master/src/yival/variation_generators/openai_prompt_based_variation_generator.py) | [OpenAIPromptEvaluator](https://github.com/YiVal/YiVal/blob/master/src/yival/evaluators/openai_prompt_based_evaluator.py), [OpenAIEloEvaluator](https://github.com/YiVal/YiVal/blob/master/src/yival/evaluators/openai_elo_evaluator.py) | [AHPSelector](https://github.com/YiVal/YiVal/blob/master/src/yival/result_selectors/ahp_selection.py) | [OpenAIPromptBasedCombinationImprover](https://github.com/YiVal/YiVal/blob/master/src/yival/combination_improvers/openai_prompt_based_combination_improver.py) | [config](https://github.com/YiVal/YiVal/blob/master/demo/configs/headline_generation_improve.yml) |
| [BERTScore: Evaluating Text Generation with BERT](https://arxiv.org/abs/1904.09675) | [Tianyi Zhang](https://arxiv.org/search/cs?searchtype=author&query=Zhang,+T), [Varsha Kishore](https://arxiv.org/search/cs?searchtype=author&query=Kishore,+V), [Felix Wu](https://arxiv.org/search/cs?searchtype=author&query=Wu,+F) | YiVal Evaluator, bertscore, rouge | [@crazycth](https://github.com/crazycth) | - | - | [BertScoreEvaluator](https://github.com/YiVal/YiVal/blob/master/src/yival/evaluators/bertscore_evaluator.py) | - | - | - |
| [AlpacaEval](https://github.com/tatsu-lab/alpaca_eval) | [Xuechen Li](https://arxiv.org/search/cs?searchtype=author&query=Xuechen%20Li), [Tianyi Zhang](https://arxiv.org/search/cs?searchtype=author&query=Tianyi%20Zhang), [Yann Dubois](https://arxiv.org/search/cs?searchtype=author&query=Yann%20Dubois) et. al | YiVal Evaluator | [@Tao Feng](https://github.com/oliverfeng) | - | - | [AlpacaEvalEvaluator](https://github.com/YiVal/YiVal/blob/master/src/yival/evaluators/alpaca_eval_evaluator.py) | - | - | [config](https://github.com/YiVal/YiVal/blob/master/demo/configs/alpaca_eval.yml) |
[Chain of Density](https://arxiv.org/pdf/2309.04269.pdf) | [Griffin Adams](https://arxiv.org/search/?query=Griffin+Adam) [Alexander R. Fabbri](https://arxiv.org/search/?query=Alexander+R.+Fabbri) et. al | Prompt Engineering |[@Tao Feng](https://github.com/oliverfeng) | - | [ChainOfDensityGenerator](https://github.com/YiVal/YiVal/blob/master/src/yival/variation_generators/chain_of_density_prompt.py) | - | - | - | [config](https://github.com/YiVal/YiVal/blob/master/demo/configs/summary_config.yml)|
[Large Language Models as Optimizers](https://arxiv.org/abs/2309.03409) | [Chengrun Yang](https://arxiv.org/search/cs?searchtype=author&query=Yang,+C) [Xuezhi Wang](https://arxiv.org/search/cs?searchtype=author&query=Wang,+X) et. al | Prompt Engineering |[@crazycth](https://github.com/crazycth) |-  | - |-  |- |[optimize_by_prompt_improver](https://github.com/YiVal/YiVal/blob/opro_implement/src/yival/combination_improvers/optimize_by_prompt_improver.py)  | [config](https://github.com/YiVal/YiVal/blob/opro_implement/demo/configs/headline_generation_improve.yml#L174)| 123213 | [Chengrun Yang](https://arxiv.org/search/cs?searchtype=author&query=Yang,+C) [Xuezhi Wang](https://arxiv.org/search/cs?searchtype=author&query=Wang,+X) et. al | Prompt Engineering |[@crazycth](https://github.com/crazycth) |-  | - |-  |- |[optimize_by_prompt_improver](https://github.com/YiVal/YiVal/blob/opro_implement/src/yival/combination_improvers/optimize_by_prompt_improver.py)  | [config](https://github.com/YiVal/YiVal/blob/opro_implement/demo/configs/headline_generation_improve.yml#L174)|
[LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685) | [Edward J. Hu](https://arxiv.org/search/cs?searchtype=author&query=Hu,+E+J) [Yelong Shen](https://arxiv.org/search/cs?searchtype=author&query=Shen,+Y) et. al | LLM Finetune |[@crazycth](https://github.com/crazycth) |-  | - |-  |- |[sft_trainer](https://github.com/YiVal/YiVal/blob/add_finetune_module/src/yival/finetune/sft_trainer.py#L40)  | [config](https://github.com/YiVal/YiVal/blob/add_finetune_module/src/yival/schemas/trainer_configs.py#L48)|

<!--  -->

